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Overview 

High-level  troubles  found  in  the  investigation 

One  example  of  process  and  structure 

Pitfalls  we  saw,  questions  to  answer,  possible  solutions: 

1.  Roles  and  Responsibilities  (technical  and  contractual) 

2.  Deliveries 

3.  Integration  Process  and  Testing 
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Troubleshooting  Integration:  An  investigation 
process  and  how  we  found  the  pitfalls 
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Troubles  We  Have  Seen: 


•  Frustration  is  high  for  all  parties  involved 

•  Integration  takes  much  longer  than  expected 

•  Resulting  product  is  buggy,  big,  and  slow;  start-up  is 
LONG 

Some  of  these  issues  can  be  traced  to  integration  pitfalls 
(and  to  classic  software  issues. ..See  the  SEI  CMMI™  too) 
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Troubles  We  Have  Seen: 

(continued) 

Release  state  is  uncertain 

•  Uncertainty  if  the  right  builds  are  in  use 

•  Trouble  tracing  requirements  satisfaction/  capabilities  to  actual 
delivered  code 

-Difficulty  in  getting  Interface  Documents  completed  and  signed  off 

-  Difficulty  in  finding  source  code  for  modules  and  in  replicating  the 
compile  process 

-  False  Fixes  caused  by  incorrect  root  cause  (due  to  version  conflicts 
between  modules) 
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Example:  Complex  Multi-Provider  Organization 
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Example:  Simplified  Complex  System 


This  schematic  represents  the 
context  in  which  the  example 
software  was  delivered 
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Example:  Interface  Build  Process  (integrating  a 
large  system) 


To  Testing? 
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To 

Module 
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Pitfall  #1 :  What  We  Saw 


•  Contracts  limited  the  access  to  various  types  of  design 
documents  and  source  code 


•  Code  was  dropped  off  in  an  immature  state  due  to 
misunderstandings  about  pre-delivery  testing 


•  Integrator  had  little  support  from  developers,  modules 
were  in  an  uncertain  state,  and  uncertain  how  to  fix 
integration  problems  that  were  encountered 
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Pitfall  #1 :  What  We  Saw 
(Continued) 

•  Providers  were  uncertain  when  to  deliver,  what  items  to 
deliver,  and  who  would  receive  reports  and  data 


•  Uncertain  of  characteristics  for  interfaces  to  other 
providers,  or  if  they  can  talk  to  other  providers 

•  Uncertain  of  the  role  of  the  integrator 

•  Uncertain  of  reuse  of  middleware  or  other  libraries 
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Pitfall  #1 :  Questions  to  Answer  and  Possible  Solutions 


What  standards  must  I  meet  for  interfacing  (internal  to  project,  and  to 
external  services)?  Use  recognized  standards  and  methods,  define  model  data  types 
and  methods  to  be  used.  Systems  should  rely  on  middleware  mediation,  interfacing  via 
middleware  and  standard,  not  directly. 

See  also  [Bianco  et  al.  2007]  for  other  considerations. 

What  do  I  have  to  provide  as  a  supplier?  (Documents?  Source  Code?)  Will  I 
have  to  let  other  suppliers  see  my  source?  NDAs?  (See  Pitfall  #2)  Supplier 
contracts  and  specifications  must  be  specific.  Ensure  NDA  signed  off  before  first  delivery. 

Source  code  access  may  be  required  if  compile  is  by  integrator  or  if  customer  will  be 
maintaining  the  code.  See  [Morris  et  al.  2010]  sec  3.1 


What  resource  restrictions  do  I  have  to  meet,  under  what  conditions,  testing 
using  what  methods,  and  with  what  other  software  running?  (See  Pitfall  #3) 

Very  clear  performance  parameters  by  service  must  be  defined  before  first  delivery  to 
integrator,  and  spell  out  testing  required  to  prove  fit. 


Note  [Lewis  et  al.  2008],  Sec.  A5  describes  a  set  of  similar  questions  to  answer 
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Pitfall  #2:  Provider  Deliveries 

Where  do  software/service  providers  deliver  code? 
What  do  they  deliver? 
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Pitfall  #2:  What  We  Saw 


Procedures  for  delivery  and  expectations  for 
delivered  code  varied 


Symptoms: 

•  Confusion  by  customer:  Where  is  my  Code?  Do  the 
modules  correspond  with  the  delivered  source? 

•  Confusion  by  Integrator:  What  versions  am  I  building? 
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Pitfall  #2:  What  We  Saw 
(continued) 

One  delivery  mechanism  was  specified,  another  in  use 

•  Code  was  supposed  to  be  registered  in  a  delivery 
database,  code  ^documentation)  and  compiled  modules 
to  a  Rational  Clear  Case  instance 

•  Due  to  time  pressure,  delivery  database  was  replaced 
with  a  form,  Clear  Case  still  used  for  compiled  modules; 
Source  Code  no  longer  required 

•  Some  Providers  followed  old  process,  some  new 
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Pitfall  #2:  Questions  to  Answer 


What 

Why 

Where  do  1  make  my  deliveries?  (i.e.  Database, 
Workflow  Engine,  etc.) 

What  forms,  standards,  process  must  1  follow 
(ex:  naming  conventions)? 

Do  1  have  access  and  to  what? 

This  is  very  important  for  configuration  control, 
and  contract  completion.  Can  also  be  useful  for 
development  in  future  releases  and  testing. 

Do  1  deliver  fully  compiled  modules? 

Even  if  integrator  does  the  compile,  need 
something  to  check  against. 

Do  1  deliver  files  (beyond  source)  needed  to  do 
a  full  compile?  (ex:  generating  scripts) 

Even  if  integrator  does  not  compile,  useful  for 
troubleshooting,  and  testing  in  the  next 
increment  (if  contracts  allow). 

Do  1  deliver  testing  stubs?  Test  Scripts? 
Instrumented  code? 

Allows  integrator  to  test  the  services  and 
module,  pre-integration. 

Do  1  deliver  documents  and  models?  What 
statistics  and  data? 

Quality  Control,  Process  Metrics,  and 
troubleshooting 

Reuse  items:  COTS,  GOTS? 

Recommended  that  the  integrator  pull  these,  not 
the  developer.  Can  lead  to  version  issues  and 
copyright  concerns. 

What  testing/certification  has  to  be  completed 
before  delivery? 

Need  to  know  the  maturity  of  the  module. . .  is  it 
mature  enough  to  integrate,  test,  deliver? 
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Pitfall  #2:  Questions  to  Answer  and  Possible 
Solutions 

Where  do  I  make  my  deliveries?  (i.e.  Database,  Workflow  Engine, 
etc.)  What  forms,  standards,  process  must  I  follow  (ex:  naming 
conventions)? 

Agreements  (and  manuals)  specify  the  process  and  specific 
locations/systems,  naming  standards,  accounts,  to  deliver  each  type  of 
file,  form,  and  dataset  (including  models  and  test  data).  Training  before 
first  delivery  for  providers  and  integrator.  Know  database  instance  for 
delivery,  and  configuration  control  applied. 

Do  I  have  access  and  to  what?  Allow  access  to  material  required  for 
developer  testing  and  integration  (compile  &  make).  Users  will  need 
licenses  in  the  delivery/CM  system.  Provide  access  to  problem/trouble 
ticketing. 
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Pitfall  #2:  Questions  to  Answer  and  Possible 
Solutions 


What  do  I  deliver?  (e.g.,  compiled  modules,  files  beyond  source 
needed  to  do  a  full  compile,  testing  stubs,  Test  Scripts, 
Instrumented  code,  documents  and  models?  Reuse  items:  COTS, 
GOTS?) 

Provide  code/scripts/files  required  for  compiiing/integration.  Provide  a 
successfully  compiled  image  to  confirm  integration.  Provide  stubs  and 
test  items  to  check  the  modules  function  once  compiled,  and  documents 
to  ensure  functionality. 

See  [Lewis  et  al.  2008],  Tables  12,  13  may  be  useful. 
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Pitfall  #3:  Integration  Process  and  (Associated) 
Testing 

A  sub-optimal  Integration  process  can  impede  success 

•  Can  be  associated  with  Pitfall  #1  &  #2 

The  Integration  Process  should  account  for  the  varied  levels 
of  integration  testing  required.  This  will  require  access  to 
parts  of  the  integrated  software,  namely: 

•  Middleware 

•  Stubs  or  Modules  which  interface  to  other  Provider  Code  (if  linked) 

•  Common  Infrastructure 

A  check-out  step  prior  to  formal  integration  can  provide 
confidence  in  modules/code  delivered  for  integration. 
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Pitfall  #3:  What  We  Saw 


Providers  services  were  written  against  older  versions  of  services  from 
other  providers  (Check-Out  can  catch) 


Providers  were  reluctant  to  help  other  providers  troubleshoot  and 
uncertain  of  the  state  of  the  stubs  used  to  test  (MOA  can  assist) 


Sandbox  environment  assembled  for  Provider  use  at  Integrator  was 
overscheduled  (Schedule  review  can  catch) 


Performance  allocations  were  exceeded,  testing  against  performance 
was  light  and  with  low  fidelity  (Performance  Team  review  should  verify) 
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Pitfall  #3:  Questions  to  Answer  and  Possible 
Solutions 

How  do  I  integrate  my  complex  system?  Integration  process  is  associated  to  the 
Architecture  and  must  be  defined  before  middleware  and  development  by 
providers. 


How  is  my  middleware  provided  to  developers?  “his  is  a  critical  path  provider  item. 
Provide  complete  releases  with  development  kit  and  key  test  scripts,  help 
manuals,  and  design  documentation.  Provide  plenty  of  time. 


What  level  of  maturity  must  the  code  have?  How  do  I  know  the  level?  What 
testing/certification  has  to  be  completed  before  delivery?  What  issues  are  resolved 
by  this  delivery?  Compilation/Test  history,  focus  on  known  problems.  Certifications 
provide  confidence.  Mature  test  infrastructure  provides  confidence  (e.g.,  Common 
test  scripts  and  procedures,  and  common  sandbox  systems),  as  does  Third  party 
pre-integration.  Mature  Software  Version  Description  is  helpful  (e.g.,  list  of 
problems  addressed,  how  addressed,  and  what  files/databases/initial  datasets  are 
required  for  the  module) 


[see  Bianco  et  al. ,  2008  for  Service  Level  Agreement  (SLA)  information  ] 
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What  should  I  deliver  to  providers  to 
allow  development? 


Middleware 

+GUI 

Infrastructure 


Provider  Y 
Service 
(Module)  B 


How  do  I  develop  and  test  this 
interface?  What  must  a  provider 
have  in-hand  to  develop?  (More  to 
an  interface  then  just  sending 
data. ..what  happens  when  I  send 

it?) 
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Pitfall  #3:  Questions  to  Answer  and  Possible 
Solutions 

How  do  my  providers  test  interfaces  to  other  providers?  What  test  facilities  do  my 
providers  have  to  mature  their  code?  Provide  the  middleware  build,  stubs  from 
other  providers,  and  a  series  of  planned  test  cases/use  cases  common  to  all 
providers.  A  sandbox  environment  with  complete  builds  for  middleware,  GUI,  and 
modules  from  other  providers  on  a  representative  processor. 

See  also  [Morris  et  al.  2010] 


How  do  I  allocate  and  test  resource  usage  pre-  and  post-  integration?  Create  key 
performance  test  cases.  Performance  is  tied  to  end-user  requirements,  which  are 
broken  down  to  specifications  that  link  to  architecture  and  design.  Provide  enough 
environment  to  show  coverage  of  expected  modules  on  representative  hardware. 
[Meyer  et  al.  2010] 


What  Scenarios/Use  Cases  with  desired  Quality  Attributes  can  be  used  to  'bleed' 
issues  out  early  on? 

See  [Bianco  et  al.  2007,  Sec.  5,  and  Morris  et  al.  2010,  Sec.  4.5] 
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Summary 

Programmatic  troubles  can  stem  from  Integration. 


Integration  can  be  a  complex  process  for  diverse  groups. 


Integration  is  a  significant  project:  Plan  for  Integration 
Process,  Deliveries,  and  Testing.  Avoid  the  pitfalls  by 
answering  key  questions  about  contracts,  expectations, 
documentation,  resources,  etc. 
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NDA 

Non-  Disclosure  Agreement 
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Software  Engineering  Institute  (SEI)  Capability  Maturity  Model 
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Service  Level  Agreement 

SOA 

Service  Oriented  Architecture 

SoS 

System  of  Systems 

SW 

Software 
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Carnegie  Mellon 


Abstract 


Service  Oriented  Architecture  based  projects  rely  on  a  diverse  team  of 
service  providers,  application  providers,  middleware  provider,  and  an 
integrator. 

Often,  the  process  for  fusing  the  elements  together  to  form  a  usable 
system  of  systems  does  not  reflect  the  procedures  that  have  evolved 
over  the  course  of  the  program,  due  to  the  immense  task  of 
communication,  due  to  timeline  pressure,  and  often  due  to  weak  process 
enforcement. 

This  talk  will  cover: 

•  a  series  of  pitfalls  encountered  on  such  large  projects 

•  to  show  the  audience  how  to  avoid  the  pitfalls 

•  and  what  processes  and  tools  should  be  considered  when  embarking  on 
integration 

•  and  what  affect  architecture  decisions  had  on  integration 
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Testing  Interfaces  in  Developing  Code:  Strategy 
-  Stub  provided  as  a  unit  by  Integrator, 

Likely  w/SDK. 

Advantages: 

•  Same  Single  Package  provided  to  each 
provider 

•  Integrator  understands  package 

•  Could  be  managed  with  one  NDA  per 
provider  (provider  to  integrator) 

Drawbacks: 

•  Models  to  create  stubs  must  be  current 
and  accurate 

•  Long  timeframe  to  create  environment 

-  May  lag  current  provider  code 

•  Very  limited  testing,  environment 
must  simulate  resource 
consumption. 


Stubs  as  an 
Environment 

Middleware 

My  Module 
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Testing  Interfaces  in  Developing  Code:  Strategy 
-  Stubs  provided  by  Each  Module/  Service 
Provider 

Advantages: 

•  Short  timeframe 
Drawbacks: 

•  Lots  of  coordination,  lots  of 
NDAs 

•  Diversity  in  Stub  currency 

•  Allows  only  limited  testing  of 
module  function. 


stub  stub  stub  stub 
Middleware 

My  Module 
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Testing  Interfaces  in  Developing  Code: 
Strategy:  Modules  Provided  by  Providers  or  by 
Integrator 


Advantages: 

•  Real  Code,  Real  interfaces:  High 
Fidelity  Testing 

Drawbacks: 

•  Complex  Planning  to  keep  current, 
avoid  serial  development 

•  Strong  Intellectual  Property 
NDAs/Concerns:  Why  arm  a 
competitor? 
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